remotemanager.connection.url module

URL base class for connecting to remote systems

class remotemanager.connection.url.URL(host: str | None = None, user: str | None = None, port: int | None = None, verbose: int | bool | Verbosity | None = 1, timeout: int | None = 5, max_timeouts: int | None = 3, python: str | None = 'python', submitter: str | None = 'bash', shell: str | None = 'bash', raise_errors: bool | None = True, keyfile: str | None = None, passfile: str | None = None, envpass: str | None = None, sshpass_override: str | None = None, cmd_history_depth: int | None = 10, landing_dir: str | None = None, ssh_insert: str | None = '', ssh_prepend: str | None = None, ssh_override: str | None = None, quiet_ssh: bool | None = True, shebang: str | None = '#!/bin/bash', transport: Transport | None = None, **kwargs)[source]

Container to store the url info for a Remote run

The url should contain everything pertaining to the _remote_, allowing Dataset to be remote-agnostic

Parameters:

  • host (str) – host address of the remote system

  • user (str) – username for the remote system

  • port (int, str) – port to connect to for ssh tunnels

  • verbose (bool) – base-verbosity for connections

  • timeout (int) – time to wait before issuing a timeout for cmd calls

  • max_timeouts (int) – number of times to attempt cmd communication in case of a timeout

  • python (str) – string used to initiate a python instance

  • raise_errors (bool) – set false to ignore errors by default in cmd calls

  • passfile (str) – absolute path to password file for sshpass calls

  • envpass (str) – environment variable containing absolute path to password file for

  • sshpass_override (str) – override the sshpass string sshpass calls

  • cmd_history_depth (int) – number of cmd calls to store history for

  • landing_dir (str) – set the directory which is treated as the ssh endpoint

  • ssh_insert (str) – any extra flags you wish to add to the ssh call

  • quiet_ssh (bool) – Option to add -q flag to ssh calls. This suppresses errors and warnings that machines can sometimes put on stderr, breaking runs. Defaults True

  • kwargs – any extra args that may end up here from a Dataset or Computer are discarded

property verbose: Verbosity

Verbose property

property user: str

Currently configured username

property host: str

Currently configured hostname

property port: int

Currently configured port (defaults to 22)

property userhost: str

user@host string if possible, just host if user is not present

clearhome()[source]

Clear the home property

Returns:

None

property ssh: str

ssh insert for commands on this connection

clear_ssh_override()[source]

Wipe any override applied to ssh. Can also be done by setting url.ssh = None

Returns:

None

property passfile: [<class 'str'>, None]

Returns the sshpass string

Added in version 0.10.2: No longer returns just the file, to facilitate overriding

Returns:

sshpass full string

tunnel(local_port: int, remote_port: int, local_address: str = None, background: bool = False, verbose: None | int | float | Verbosity = None, dry_run: bool = False) → CMD[source]

Create a tunnel to the host, between the local and remote ports

Parameters:

  • local_port (int) – port to open on the local side

  • remote_port (int) – port to access on the remote side

  • local_address (str) – change the local address of the tunnel

  • background (bool) – creates the tunnel asynchronously if True (default False)

  • verbose – override verbose setting for this call

  • dry_run (bool) – do not execute if True> Defaults to False

Returns:

The CMD instance responsible for the tunnel

Return type:

CMD

property is_local

True if this connection is purely local

ping(n: int = 5, timeout: int = 30, verbose: None | int | bool | Verbosity = None) → float[source]

Perform and monitor a ping command

Parameters:

  • n (int) – number of pings to aim for

  • timeout (int) – kill the process and return 0 if this period is elapsed

Returns:

(float) latency in ms

expandvars(string: str) → str[source]

‘echo’ a string on the remote, returning the result

Parameters:

string – string to be expanded

Returns:

str

cmd(cmd: str, asynchronous: bool = False, local: bool | None = None, stdout: str = None, stderr: str = None, timeout: int = None, max_timeouts: int = None, raise_errors: bool = None, dry_run: bool = False, prepend: bool = False, force_file: bool = False, landing_dir: str = None, stream: bool = False, verbose: None | int | bool | Verbosity = None) → CMD[source]

Creates and executes a command

Parameters:

  • asynchronous (bool) – run this command asynchronously

  • cmd (str) – command to execute

  • local (bool, None) – force a local or remote execution. Defaults to None

  • stdout (str) – optional file to redirect stdout to

  • stderr (str) – optional file to redirect stderr to

  • timeout (int) – time to wait before issuing a timeout

  • max_timeouts (int) – number of times to attempt communication in case of a timeout

  • raise_errors (bool) – override for global setting. Raise any stderr if encountered

  • dry_run (bool) – don’t exec the command if True, just returns the string

  • prepend (bool) – always attempt to use ssh_prepend

  • force_file (bool) – passthrough for CMD force_file argument

  • landing_dir – set the directory which is treated as the ssh endpoint

  • stream – Attempts to stream the output as it arrives on stdout

Returns (CMD):

returned command instance

property utils

Handle for the URLUtils module

test_connection() → None[source]

Create a ConnectionTest instance and run the tests

Returns:

None

property connection_test: ConnectionTest | None

Return the connection test object

Returns:

testing object

Return type:

ConnectionTest

property connection_data: dict

Returns the results of a previous connection test

Returns:

(dict) connection data

property latency: [<class 'float'>, None]

Attempts to access the latency property of the stored ConnectionTest

Returns:

connection latency in seconds, if available. Else None

Return type:

(float)

static download_file(file_url: str, filename: str) → None[source]

Download file at url file_url and write the content out to filename

Parameters:

  • file_url – url of file

  • filename – name to write content to

class remotemanager.connection.url.URLUtils(parent: URL)[source]

Extra functions to go with the URL class, called via URL.utils

As it requires a parent URL to function, and is instantiated with a URL, there is little to no purpose to using this class exclusively

Parameters:

parent (URL) – parent class to provide utils to

file_mtime(files: list, local: bool = None, python: bool = False, ignore_empty: bool = False, dry_run: bool = False) → dict[source]

Check file modification times of [files]

Parameters:

  • files (list) – list of paths to files

  • local (bool) – force a local search

  • python (bool) – ensure python style search is used

  • ignore_empty (bool) – also check the filesize, ignoring empty files

  • dry_run (bool) – print command only

Returns (dict):

{file: mtime (unix)} dictionary

file_presence(files: list, local: bool = None, dry_run: bool = False) → dict[source]

Search for a list of files, returning a boolean presence dict

Parameters:

  • files (list) – list of paths to files

  • local (bool) – force a local search

  • dry_run (bool) – print command only

Returns (dict):

{file: present} dictionary

search_folder(files: list, folder: str, local: bool = None, dry_run: bool = False) → dict[source]

Search folder for files, returning a boolean presence dict

Parameters:

  • files (list) – list of filenames to check for. Optionally, a string for a single file

  • folder (str) – folder to scan

  • local (bool) – perform the scan locally (or remotely)

  • dry_run (bool) – print command only

Returns (dict):

{file: present} dictionary

touch(file: str, local: bool = None, raise_errors: bool = None, dry_run: bool = False) → CMD[source]

perform unix touch, creating or updating file

Parameters:

  • file (str) – filename or path to file

  • local (bool) – force local (or remote) execution

  • raise_errors (bool) – raise any stderr encountered

  • dry_run (bool) – print command only

Returns (CMD):

CMD instance for the command

mkdir(file: str, local: bool = None, raise_errors: bool = None, dry_run: bool = False) → CMD[source]

perform unix mkdir -p, creating a folder structure

Parameters:

  • file (str) – name or path to folder

  • local (bool) – force local (or remote) execution

  • raise_errors (bool) – raise any stderr encountered

  • dry_run (bool) – print command only

Returns (CMD):

CMD instance for the command

ls(file: str, as_list: bool = True, local: bool = None, raise_errors: bool = None, dry_run: bool = False) → [<class 'remotemanager.connection.cmd.CMD'>, <class 'list'>][source]

Identify the files present on the directory

Parameters:

  • file (str) – name or path to folder.

  • as_list (bool) – convert to a list format

  • local (bool) – force local (or remote) execution

  • raise_errors (bool) – raise any stderr encountered

  • dry_run (bool) – print command only

Returns (CMD, list):

CMD instance for the command, or the list if as_list is True